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^FOREWORD 



In his rejport- for the Commonwealth Tertiary Education 
CommissioQ,' Academic Development Ufiitrs, Professor Richard 
Johnson said, in, reference to knowledge about educational masters: 

There exists a substantial body of l<no\yledge 
on many of these matters and there ar^ 
techniques for finding* answers to the 
questidhs. Howeveri just because the bpdy of 
knowledge is substantial, just 'because the 
processes of higher education are complex, it 
cannot* be expected that the academic 
pursuing research and teaching in another 
discipline or the administrator coping with 
the day-to-day and year-to-year urgencies of 

• an institution can master or keep up with the 
range o,f contemporary thinking and' research 
results. \ 

While that is certainly cci-rect it is also true that most 
staff in" tertiary education are interested in carrying out their 
various tasks and responsibilities well. 

To this end m^ny of them would sppreciate some guidance 
on one or other of these tasks and respionsibilities, But staff do not 
have time for a detailed study of the literature on these issues. 
The other reason for the frustration, of staff who go in search for 
guidance is that many of the publications available are written by 
people who are conscious of the academic tradition of writing and 
hence prepare treatises that attempt to be domprehensive, argged 
in detail and fully documented. This is not what staff need ^ they 
need ^sorne idea^ and pointers that they can relate to their 
particular circumstances and some ^ brief guidance to further 
reading if they so desire^ 

• * • 

• The publications in this HEROSA Green Guide Series are 
explicitly designed to meet these needs. They are 4*elatively short 
(about 30 pages), inexfJensive, easj^ to read, and concentrate on 
supplying ideas rather than a fully argued comprehensive cover of 
an area. Their bibliographies are deliberately not comprehensive 
but are designed to be helpful - these are not academic studies of 
an area but, as the series title indicates, guides. 

Each Guide is designed to cover one aspect of a staff 
members tasks and responsibilities. Each Guide has beon 
commissioned by the HERDSA Publications Committee and 
refereed but the content is the responsibility of the particular 
author(8) and does not necessarily represent the views of the 
Committee or the Society. 

We hope that staff will find the Guides useful and welcome 
comments on individual Guides and on other areas that could be 
covered by simitar volumesc 



HERDSA Publications Committee 
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I INTRODUCTION ' 



What is a departmental review? * 



Universities and colleges have always b^en involved in a 
variety of reviews and evaluations* The most visible forms of 
review are the regular accreditation of courses in colleges of 
advanced edu'^ation, the evaluation of professional courses by th'? 
relevant professional societies, and the evaluation of individuals 
throughout higher education for purposes of tenure and promotion. 

More recently a certain scepticism about the value of 
higher education, coupled with declining participation rates and 
some criticism of thTe quality of higher education, has led 
governments to impose financial constraints. This in turn has led 
to a new type of 'review*: for example, enquiries as to whether 
particulafr institutions should be closed or amalgamated, whether 
certain departments should be closed down, whether staff should be 
transferred from one department or institution to another, and so 
on. 

There are many views as to what constitutes a 'review* but 
this booklet Is not about any of the above. We are concerned with 
departmental or school reviews* which involve a systematic 
investigation of the structure and functions of an academic 
dppartment with a view to evaluating ^and improving its 
performance. The word 'systematic' is significant. It implies a 
properly planned and responsibly conducted operation. A 
departmental review could consist of both self-critical examination 
within the department and review by a largely external panel, with 
both activities systematic. Any changes recommended would be 
designed to improve the quality of a department's work by pointing 
the way to improved performance. Sometimes, because a 
department was already functioning extremely well, * changes 
recommended would be very minor; occasionally, they might be 
substantia!^' even fundamental. It would be unpsua^ to find a 
'perfect' department. 

In defining a departmental review it rRay also be helpful to 
say what it is not. U is not a backward->looking evaluation confined 
to judgments on past performance\departments are living entities 
and any judgments about them should be interim and forward- 
looking.^ Nor is a de;)artmental review an appropriate occasion for 
detailed e*va!uation of individual staff performance. It will 
inevitably have some implication^ for the work of individual staff 
< 



*We use the general term "departmental review" cO refer to a 
review of any academic unit: department, school, institute, centre, 
or unit. 
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^ members: their work in aggregate^ is the department's work. In 
' pointing to any implications for individuals, a review panel should 
tsgerate on the same * forward-looking, constructive principle 
referred to above, and must at all times have due regard for th^ 
rights of individualst « 

Why is this booklet needed? 
« 

Properly carried' out, departmental reviews are 
constructive exercises that contribute handsomely to forward 
planning in departments. They can constitute an inteli|gent and 
thoughtful response to changing demands, changing situations, and 
the changing fortunes of higher education. 

But this can only happen if they are properly conducted. 
We have seen examples of departmeiitat reviews which consist 
'entirely of a series of 'data-free' judgments made merely on the 
basis of the prejudices of members of a review panel; of reviews 
which have r/ianaged \ to alienate the entire staff of the 
department; and of reviews that have not diagnosed any of the 
significant ills of a troubled department. 

This booklet is written to help those involved in review^ - 
members of review panels, members of a department being 
reviewed, and administrators - to carry theiji out in a professional 
manner. It has baen prepared as a guide and introduction only, arid 
is not intended as a universal prescription. Many of the details of 
how a review is conducted will depend on the si^e of an institution, 
whether it is a college or a university, and the size of the 
department reviewed. A much more complete .treatment is 
available in Roe, McDonald and Moses (1985). * 



Are departmental reviews a good idea? 

There are those who would suggest that systematic 
evaluation of departments should not be carried out. The reason^ 
that car).be advanced to justify this view are familiar to us all, and 
are cov^^^by Roe and McDonald (1984, pp. 9-13). They include a 
fear thH^jliose doing the evaluating wilt not be competent, or 
sufficienUy sympathetic, or that they will use the wrong criteria; 
or that re\oews cost money and don'f necessarily serve any useful 
purpose, *and may actually do harm. Staff associations are also 
concerned that reviews could be used to further erode the situation 
of academic staff in higher education and could divert time and 
energy from more important acadeTnic pursuits. 

It is true that a badly conducted review may indeed fulfil 
the legitimate fenrs expressed above.^ But the benefits of welU 



conducted reviews have been substantiaU We and our colleagues 
have seen examples of departments given a ne>y lease of life and 
opportunities to pursue new directionSf of individuals who became 
aware for the first time that their value to a department was 
recognised, of improved organisational arrangements, of better 
teaching arrangements, and other'benefits arising from a serious 
introspective examination of activities which may have proceeded 
unquestioned for some time. Reviews can also provide valuable 
input into decisibns on institution-wide planning policies. 
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n PLANNING A REVIEW 



1 What Is the purpose of the review? 



There is a wide variety of possible reasons for reviewing a 
department^ ranging from the desire of its own members to obtain 
Information which will enable the;D to improve that performance 
to an ins^titution's attempt to justify a decision already made to the 
detriment of that department. The latter reason for a review i.s 
morally indefensible. Indeed, a review which simply passes 
judgments on a department is difficult to justify. If judgments are 
made about the quality of a department^ the institution has an 
obligation to provide encoura^ementi opp*ortunities and facilities 
for any defects to be remedied: departmental reviews should 
manifest the important link between evaluation and development 
and should, wherever possible, point the way towards improved 
performance. 

Those who have written on the subject emphasijse the need . 
for reviews to have an honest, constructive purpose. The central 
pjjrpose should be seen to be Improvement, not judgment: 
identifying priorities for development, planning for the future, and 
reassessing existing commitments and activities. 

It should also be noted that for the ^staff of the 
department, the benefit of the review lies in the propess - thq 
preparationi data co*il8Ction| discussions, and questioning - as much 
as in the final report. ' 



2 What are the terms of reference? 

t 

The scope of departmental reviews can vary consi^lerably. 
However, it is common for reviews to be concerned with a 
department's course offerings, teaching, research/professional 
activities, staffing, administration, and future needs. 

Terms of reference would normally include the following, 
adapted to suit each particular institution: 0^ 

i The appropriateness, quality and balance of the 
curriculum, and the quality of teaching. 

i'l The appropriateness, quality and balance of- research 
development/professional activities carried out within 
the department. 

iii Whether accommodation, facilities, allocation of 
teac'ning/research/equipment funds, internal 
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administration, institutional policies and 
ad minist ration, support structures and staffing 
arrangements are serving the department and the 
* students adequately. 

iv Whether there Is*' adequate liaison between this 
department and cognate departments, the rest of the 
institution, and the relevant outside bodies. 

As a consequence of their deliberations, a. review panel would 
normally be &3ked to report on: 

V Whelher the department should continue in its present 
form, and whether any changes *^hould be made to its 
structure. 



i Desirable future directions for the department. 



« A review panel would normally be asked to report on those 
rriatlers, taking into account natjonal and international standards in 
the discipline or profession. It would naturally be expected that 
they* would go beyond mere judgment and would suggest 
improvements wherever app'ropriate. 

In odditfon to the above fairly standard terms of reference, 
there wfill be others ''which are somewhat more evanescent, 
appearing as certain concerns become fashionable and, in time, 
43eing repfaced by others. A current examp4e would be to request a 
paneMto suggest how any new development^ can be funded by 
contractions elsewhere in the department. Tnere will also be other 
items specific to the institution or to the particular department: 
for example, interactions between different disciplines or areas, 
whether a department was preparing its graduates ac^equately for 
the profession, or the role of departmental seminars. 



It is inevitable that in the course of a review, the panel 
will form opinions about the members of staff who comprise the 
department, and in ^ome cases these opinions will affect their final 
fi^ommendations. However, the review of individuals as members 
of a department, in order to mjake judgments on that department» 
should not be confused with the in-depth evaluation of individuals. 
It would not normally be appropriate for a departmental review to 
duplicate an institution's proced»;res for reviewing staff 
performance. * 
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How will it be organised? 



A number of key questions*- occur under this heading: vyhat 
machinery exists or must be established for conducting the 
review; vyho will carry it out; what information will be sought^ 
from whom, and how* These questior^s are addressed on the 
following pages. 

• V 

Special attention needs to be paid to the provision of 
administrative support; the requirements will differ for each 
reviev/. But gathering and summarising information, preparing for 
meetings, attending meetings and assisting in, preparing the draft 
and final reports m'ight take the equivalent of a month's work by 
someone in a senior clerical position. This provision needs to be 
consciously made, for if th^ support is skimped, if the review 
committee does not have the Information it needs in the form it 
wants, its discussions will suffer* ^ 

Some general principles which shoijld be observed in \he 
organisation of a review are given in Section V. 



A Who will do it? 

' There is a wide range of possibilities, from a single 
evaluator or team entirely from outside the institution, to a 
working party drawn entirely from within the department itself. 
The composition of a review panel wilt depend on the institutional 
context and the purpose' of the review. There is no general 
formula, but we would argue for a panel that contains 

the necessary expertise in the subjectls) and the profession, 

knowledge of the working of the institution, and * 

competence in principles and methods of evaluation. 

There is a considerable difference of opinion as to how the 
above requirements are best met in a panel which needs tu remain 
a workable size. We would argue for the inclusion of the first four 
categories below, but arguments can be made in favour of every 
category. 



i. Experts in the subject(s) from outside the department 
and probably outside the institution. 

\ 

ii. Representatives of the professron (if appropriate) 
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ERIC 



ui. A senior administrative officer of the institution, as 
secretary to panel ' 

iv. A specialist in evaiuatlon, either as a full member of the' 
pane! or consultant. (Such specialists can help the panel 
plan the review so as to ensure that maximum benefit is 
derived from itj they can advise on procedures, , 
methpdology and give other practical advice, and they 
can even gather, analyse and present data.) 

V. Members of staff of the institution (outside the 
department) 

vi. Members of ihi department 

• * 

vii. Chief Executive or Deputy . ^ 

viii. Students or graduates. 

The arguments that a substantial proportion of a review 
tram should be external are based on the need for objectivity, 
impartiality, and fresh perspectives. The arguments against relate 
mrtifily to the emphasis placed on the value of self-study. Some 
\vmild claim that an institutional ethos where evalug'ion is 
initiated internally and carried out responsibly in an atmosphere 
whnre both departmental autonomy and honest jself-scrutiny are 
priced is the desirable ideal. But even an intqfrnally controlled 
dppartmental review is likely to make use of one or two 'outsiders', 
for the reasons given above. 

Most review panels are a compromise, with external and 
iniornal members in various proportions. The question of 
represent&tion is also a complex one. !f local tradition demands 
Ihnt oil interested parties should be directly represented, the panel 
may be too larae to be efficient'. If reviews are In two stages (with 
a review pan^ operating at the second stage, preceded by a 
departmental self-study) there may then be less need for 
substantial departmental membership of the panel. Indeed, more 
Important than actual departmental representation is an 
arrangement that ensures that a review has considerable 
interaction between the parrel and staff of the department. 



5 What information, from whom, and how? 



In carrying out a departfnental review, a key issue is 
deriding what information should be ot?tained, and this will, of 
cnursQ, be dictated by the terms of reference. Some ihformotion 
will be common to most reviews; for example, recent annual 
reports, information about staff, information about courses 
offered, the departmental budget, data on student enrolment and 
graduation rates, publication records, and so on. However review 
panels or evaluators are often tempted to request much more 
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information than they can effectively use. Clarification of the 
purposes of any particular review wilj help ensure that this does 
not happen, - 

On the other handy there are enormous dangers in the 
opposite extreme: «the data«free reviev/. There are many examples 
of reviews based on brief visits, opinions and impression^ without 
the benefit of anything but the scantiest informalion. However 
^iich reviews are increasingly chaUenged and are becoming less 
Hkely. 

The skill, in short, is in using a sufficienC amount of the 
• right sort of information. In the case of a comprehensive review, 
such information could be selected from the following list: 

U Philosophy, goals and image of'a department 

the department's goals, how its activities relate to 
its goals, r.he weight of "profespionaP versus 'general* 
education, the department's image both* within and 
outside the institution, etc. 

ii. Academic staff and staffing 

* numbers, ranks and experience of staff, research, 

I professional and other interests, provisions lot staff 
^ induction 'and development, policies and procedures on 
staff evaluation, etc. 

iii. Students 

admission standards, number of students 
applying/admitted, data on progress, performance, 
transfer and dropout (for botti undergraduates and, 
where appropriate, postgraduates), reasons for dropout, 
s^etc. 

iv. Curriculum, teaching, assessment 

overaii goals, structure (including integration of 
different subjects and between different areas), depth 
of treatment, contact hours, balance between theory 
and practice, texts and other teaching materials, 
quality of teaching, adequacy of assessment, quality of 
supervision of research students, provision for ongoing 
evaluation and monitoring, etc. (A detailed treatment 
of these issues Is contained in chapters 2 and 3 of Roe 
and McDonald (r984).) 

v. Research 

individual research projects; team research, 
integration of postgraduate students into team projects; 
allocatiun of research funds across projects, research 
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grants, areas in need of further research, etc. 

vi. Re3our6es, facilities and organisation 

... expenditure of funds, criteria for distributing funds, 

costs per student, physical facilities, equipment, 

teaching support services, organisational and 
committee structure, provision for planning, etc. 

vli. The future 

... staffing constraints, possible flexibility, lit<ely 
developments in the department's field, possible 
changes in st^udent demand, etc. 

The simple answer to the question of 'from whom?' is 'all 
parties with a legitimate interest'. In a departmental review, the 
h'^ad, staff and students of that department, graduates, deans, 
central administra^tors, professional organisations might all be 
sources of information. The methods used should be appropriate to 
the nature of the questions asked, and the number of people 
involved. In practical terms, relatively restricted categories of 
people, such as department heads, deans, senior administrators, are 
more likely to be interviewed, while questionnaires may be more 
practicable with the (presumably large) body of students. 

Decisions on ail three questions • what ir.formation, from 
v/hom and how - are more likely to be the right decisions, as far as 
the ultimate effectiveness of the review is concerned, if they are 
made in the context of continual reminders of the purpose, focus, 
scope and terms -of reference of the review. The point is so 
obvious that to state it is almost embarrassing. Yet it is not 
uncommon for the hunt for information, once it has begun, to 
develop a kind of wild momentum of its own. Eager evaluators 
need to pause sometimes in the excitement of the chase and 
reflect on why and where they are running. 



How will the information be processed? 



It sometimes happens that much of the information 
available (already documented) or gathered for a departmental 
review Is not really analysed at all. It rather approximates a 
patchwork quilt of extreme variety and complexity, which Is 
apprar^ed by the review (tanel; they reach their conclusions by 
impression rather than through analysis. This is a risky procedure, 
resting on the dubious assumption that a number of subjective 
Impressions results ^ in an objective judgment. Personal 
interpretations of the data, undue concentration dn aspects which 
happen to patch an individual's attention, are inevitable in 
departmental reviews. It would be unrealistic to expect 
otherwise. Nevertheless, such bias can be kept to a minimum if 
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there is sufficient determination - and sufficient resources - to. 
analyse properly the data which deserve analvsis and are capable of 
it. . ' 

Data analysis requires considerable . expeTtise and 
experience if it la to be carried out professionally, and any 
treatment of it is bevond the <JCope of this booklet. The following 
•guidelines' are offered as points which review panels might like to 
ponder: 

U There little solace in completing a briUiant data 
analysis if the wrong problem is being addressed. 

ii« Those who conduct reviews and those who intend to 
make decisions based upon them will benefit greatly if 
they plan data analysis together. 

iii. There are usually many,/ather than few, useful ways to 
analyze data collected as part of an evaluation. 

Iv. Ethical issues have a way of turning upjn data analysis. 

v. Take care when moving from description to association 
to prediction. 

vi. Expect the data analysis as it unfolds to guide you; 
don't overplan. , 

(I * 

vii. Data analysis hqs a way of taking longer than 

anticipated. 

A full treatment is contained in Winston (1984), from which 
the above list is adapted. 



7 What kind of report will be made, and to whom? 



Reporting follows naturally from the collection and 
analysis of data and the^ exercising of professional judgment by 
members of the panel. The possibility that lata may be collected 
and analysed to bolster a report already written (at least in 
someone's mind) has already bpen acknowledged and deplored. 
Aside from such hidden agendas, one Twould expect a report to 
present the data as fully and faithfully as possible, and to interpret 
the data with both insight and objectivity. In all this, the 
evaluators follow the conventions observed h*' researcher^ in 
writing a research report. 

There are, however, some significant differences. The. 
most important is that m most cases the report of a review panel is 
meant to have immediate practical consequences. Those 
conducting a departmental review may still see their responsibility 
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as very limited} as with many research repoits, the datB may be 
presented, -cautiously interpreted if at all, and rea^ders left to 
supply their own conclusions and recommendations. At the other 
extreme, the report may have authoritative conclusions and firm 
recommendations, so that those* conducting the review have 
assumed a considerable responsibility for bringing about change in 
the department. , 

Another signiflcarit consideration in reporting on a 
departmental review is the audience (or audiences) to which it is 
Addressed. The question of how it originated Is obviously relevant 
here, together with the context in which it is taking place. , A 
review report may be to the head of the institution, to a facujty or 
school, to the department head, or to all members of the 
department. Often the evaluator, even though addressing the 
Institution's h^ad, will have members of the department very much 
in mind, as reading, acting upon, potentially profiting from, the 
report. ThCis they constitute in some senses a second audience. 
Everyone with a legitimate interest in the report is a potential 
audience, and it would be normal practice for all such Individuals 
and groups to^ee the final reporti ^ ' 

• * 

8 How will implementation be facilitated? 

t • 

The first steps towards implementation should be taken 
long before the report is written. In the very planning of a* 
departmental review, the prospects for implementation of" 
whatever report emerges should be considered and measures taken 
to enhance those prospects. Such measures include creation^of a 
favourable climate, informing and consulting members of the 
department wherever possible, and ensuring a level of Commitment 
to the review so that many will have at least an expectation that 
the report will be implemented, and at best-a personaJ ttake in its 
implementation. . , « ^ 

Other moves likely to assist implementarion involve 
directing diflerent aspects of the report t<f the 'right' audiertces, 
inviting interested parties to comment on the report in its draft 
stage, and in general doing everything possible to ensure tMt the 
report will be favourably received. ' , 

The conservatism of most educational institutions makes it 
imperative that there are explicit procedures for the 
implementation of reports: the possibilities for the adoption of 
delaying tactics and of making only cosmetic changes are too great 
to leave Implementation to chance. Some person or group needs to 
be Iflentlfied as having the responsibility for carrying out 
recommendations, arid there should be time limits for action; it 
may be that deadlines for each stage are feasible and/or a report 
on the implementation may be required after, say, two years. This 
report should be accompanied by a copy of the original review 
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report so that all parties can see what has been done and what left 
undone. 



This is not to suggest that every recommen'*ation of every 
review committee should always be implemented. A decision 
against implementing a report may be taken because it will cost 
too much, in time, money or other resources, or the advantages 
likely to be gai:ied are judged as insufficient to justify the cost, or 
bepause making a change for the better in one place may have as 
an ineviteble consequence a change for the worse elsewhere, or 
because it might not be capable of implementation with existing 
staff, or because in sume way the proposed change is Incompatible 
in policy or philo^sophy with broader institutional considerations. 
There are these arid no doubt other respectable reasons which may 
result in a failure to. implement. Whenever they are advanced, 
however, they should be subjected to rigorous examination, lest 
they are being used simply as. excuses for inaction by thosQ who 
«;have other less convincing reasons for resisting the implementation 
of a review report. * 
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m STEPS IN CARRYING OUT A REVIEW 



In undertaking a departmental review^ what should one 
actually do? " In this section a number 'of steps, in plausible 
seqi^ence, is suggested. The steps should be treated as suggestions 
or guid^elines rather than prescriptions. There will be good reasons 
In particulfiy contexts for omitting or combining steps or 
interposing otHeia. 

, One important issue pfesent throughout the planning and 
conduct of a departmental review is who Is to be responsible for 
what. Clarity on this question is essential, although not all steps in 
carrying out a review will necessarily be taken by the same 
individual or group. Reviews have often failed in one or more 
important particular because clarity vyas lacking. 



Appoint a review panel 



This first step will be taken by whoever has overall 
responsibility for the review: the chief executive of the institution 
or its academic board or by a committee which is charged with the 
overall supervision and monitoring of a programme of departmental 
reviews* for the whole institution; or, in the case of an internally 
controlled review, by the department itself or its head.. (If there is 
to be an overseeing committee, appointment of jts members Will, 
of course, bp Step 1). ' n 

Some comments on the possible constitution of^a review 
panel are given in * 



2 Take the decisions listed in Section 11 above 

This - the entire planning process • is obviously a very 
large step, involving several hours of concentrated work by the 
review team. Although the early taking of all th^ listed decisions 
is strongly advised, the team may find reasons to postpone some of 
them or make only a general and tentative decision at first* 

Not all t'he decisions need be the responsibility of the 
review team. If, for example, there is a supervisory committee as 
noted above, it may take the decisions on purpose, scope, terms of 
reference, and even on ensuring implementatioh, before a review 
team is even appointed. The nature of the review report and its 
audiences may also be treated as matters of institution policy and 
decided by its academic board and/or chief executive. 

Therefore, depending on the authority structure with 
respect to reviews, the two above steps may involve additional 
steps, and a variety ofo individuals or groups may be involved in 
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taking them. The important thing !s that, by this point| a team for 
carrying out the review has been esiablished; it has clearly defined 
responsibilities; and all significant planning decisions have been 
taken (by someone) and are cle&>*ly understood both by the review 
team and by those who are to be reviewed. It is then ready to 
proceed to Step 3. 



, 3 Make the terms of reference operational 



The terms of reference may be expressed in such specific 
detail as to make this step unnecessary. Usually^ iiowever, terms 
of reference are general,, particularly so if they represent a policy * 
determination for aU^ reviews to be carried out in the institution 
(see for example 11.2). Examples would be 'Assess the academic 
quality of this department'^or 'Decide whether the department 
should continue in its present form and whether changes ar.^ needed 
which would increase its effectiveness.' Faced with such latins of 
reference, it is an important* task for the review team to turn them 
into specific and researchable questions. If asked to decide 
whether a department should continue in its present form, the 
Committee has to consider what it needs to know in order to be 
able to make that' decision, what aspects of the present functioning 
of the department it will study - in general to translate its broad 
brief into practical, action-ready terms such as will facilitate the 
steps which follow. 

How the committee does this will depend on how they 
intend to go about their task. One possible approach is to identify, 
as quickly and easily as possible, the main 'issues' in the 
department, and to then concentrate on those. But first the 
committee would need to satisfy themselves that they know enough 
about the department to decide what- those issues are. 



, 4 Locate exis.ting documentation relevant to each term of 
reference 



Thi» terms of reference are now in-^a form which in effect 
has identified the key issues to be addressed in this particular 
review. Much of the information relevant to those issues will 
already documented somewhere. But it will need to be located 
and collected. One term of reference might be 'Trends in course 
development, student enrolment, and student performance in the 
department over the last ten ' years'; and all the relevant 
information can, in principle, be retrieved from department or 
institut*'-*'^ records. ^ 
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Decide what other information is needed 



• When the existing documentation has been located, it will 
have to be assembled (See Step 6). At the same time, the review 
team will be deciding what additional Information it needs for each 
term of reference. This step is likely to involve the team in its 
first serious conflict between ideals and reality. The decision^j 
provisionally taken in Step 3 as to what it needs to know may now 
appear unduly 'optimistic. Faced with the imminent task of 
collecting the information^ and perhaps somewhat daunte*d by a 
potentially huge quantity of existing documentation, the team's 
determination as to what It needs to know may weaken a little. 



6 Assemble existing documentation • 



• Again, close to the realities, the team may opt to be 
somewhat selective rather than comprehensive. At least it^will 
expect those' providing the material to provide it, in summary form 
jvherever this is practicable. It should be noted, however, that this 
may involve several steps, depending on the context. For example, 
a review team, particularly if it is doing its own information- 
gathering, may need to give* quite specific instructions to those 
providing information as to what to provide and in what form; or it 
may need to have a preliminary look at samples of the material 
available before it can give such instructions. If the review team 
has a research assistant, it may give him/her detailed briefing 
ftnd/or leave to him/her certain decisions about what to get and 
how. Thus, although 'Assemble existing documentation' is listed 
here as a single step, the assembling of what the review team will 
actually use, and in*usable form, may be a series of operatioms. 



7 Establish procedures, . devise instruments, etc. for 
collecting other information. 



As indicated in the introduction, this sequence of steps 
. may be only approximate. Step 5 - deciding what information not- 
already documented will be needed - may be taken partially after 
Step 4 and partially after Step 6. It is also possible that Steps 5 
and 7 will be repeated, perhaps more than oftce. In other words, 
just as study of the existing documentation will suggest a need for 
other information, so will study of that other information lead to 
t^^*? identification and collection of further information. The need 
for an additional stage of information-gathering may become 
apparent either before or after -any one set of information is 
processed, or indeed both before and after. 

However, each time the review team decides to seek 
further information it will need to establish procedures for getting 
that information; or, if it is fortunate enough to have a skilled and 
experienced research assistant, request that person to devise the 
means of obtaining it. 
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^ Information for a review panel 



Somq examples of the sort of information a review panel might use 
are given below, although the list is not intended to be 
prescriptive. Note that some of the information would normally be 
already documented but other items would need to be prepared or 
gathered* 



i* Statement of the institution's policies and procedures 
for reviews, including the reasons for this department 
being reviewed at this time 

ii. A briefing paper on the background of the department/ 
organisation! current ctatus, and anticipated future 
development ' 

iii. Copies of departmental handbooks and annual reports 

iv. Df^partmentai bud.get 

v. Academic staffing - establishment, present members 
(names, ages, levels, areas of academic interest, c,v/s) 
- and level of support staff 

vi* Information on courses offered by the department • 
course structures, requiremerjts, course outlines, 
teaching and assessment methods 

vii. Information on calculation of tej|ching loads 

viii. Data on enrolment patterns over the last five years, 
studenttstaff ratios, graduation rates, progress and 
dropout data (plus, possibly, comparable data from 
other institutions) 

» 

ix. Pass and grade statistics of courses in the department 

X* Information on research in the department - projects, 
publications, other output, grants received. 

xi. Results of evaluations of Selected or key courses 

xii. Table of destinations of graduates from the department 
and in the case of professional courses, how this relates 

' to the department's intentions, 
« 

xiii. Reports on any specific area of interest - e.g. student 
workload, effectiveness of prerequisites, reflections of 
graduates, reasons for student withdrawal 
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8 CoUect this information 

i 

The size of this step will vary greatly accprding to 'the 
tehms of reference of the departnientat review. Where a review is 
largely confined to matters on which adequate documentation 
already exists, additional enquiriesr will be few. At th other 
extreme, 'collecting the information' may involve lengthy 
questionnaires to students, graduates, employers, members of a 
profession, together with extensive interviews, conducted by or on 
behalf of the review panel, of many individuals, including all staff 
of the department under review. * 



9 Analyse or otherwise process the information 

This step, to t-^al with information specially gathered for 
the review, parallel^ Step 6 with respect to existing 
documentation. The purpose in each case is to get the data into a 
for^ in which it can be written up as a report. This may mean, in 
the case of quantifiable information (such as that collected by 
questionnaires), production of tables. Data from interviews may bo 
recorded, transcribed, or m^re probably summarised in some way. 



10 Interpret the data 



This vitally important step may take place over a number 
of meetings and discussions by the review team. It is the process 
of passing from description to evaluation. Th^ review team has all 
its information in some organised form, information which is, .in 
essence, a detailed description of the department's functioning. It 
will most likely also Include'a nunjber of individual opinions obout 
the department, which are evaluative In nature. But the review 
team is now in a position ta do jtsi evaluation: to ask Itself 'What 
(Joes it all mean? What conclusions ahould we draw? What weight 
should we attach to these facts and to these opinions?' and other 
such questions. The point has arrived at which the review team 
must exercise its informed professional judgment about the 
department. 

In interpreting the data and formulating recommendations, 
members of review panels should resist the temptation to put their 
personal stamp on the department; they are being asked to review 
a department, not to say whether they personally would prefer to 
do things differently. 

That small piece of advice is as far as we can go in a short 
handbook in directly assisting the process of judgment formation. 
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We assume that a thoughtfully constituled review panel containingi 
say, some biochemists will be able (following such plans and steps 
as those outlined here and employing their own academic qualities) 
to arrive at informed professional judgments about a biochemistry 
department. 

♦ •• 

In some reviews the time will' come when it is necessary to 
face difficult issues squarely. Professor Richard Johnson, who has 
chai^d many reviews at the Australian National University, speaks 
of Mancing the boil' : the committee members have learned to trust 
each other, somebody gets impatient and decides to speak openly, 
and with a great sense of relief - the committee begins to come* 
to grips with the real problems of the department and the measures 
needed to improve the position. An experienced chairman can 
anticipate and expedite this process* 



11 Draft a report j ' 

A review team may approach this task in several different 
ways, partly dependi/ig upon how it intends to structure its report, 
partly upon who is going to write it. Some sections of the report 
may indeed be drafted quite early (as early as Steps 5 and 6); fur 
example, on the history of the department or comprising other 
self-contained descriptive information which is already 
documented and where no further enquiries are to be made. On the 
other hand, the team may decide to v/rite nothing (except 
members' own notes) until everything is in front of them, and the 
task may then be divided between team members/ each talcing 
responsibility for a section, or undertaken by the chairman after 
lengthy discussidhs as to what should be included in the report. 
One, two, or even more drafts may be produced and considered by 
the review team before there is a draft regarded as appropriate Jor 
more public scrutiny (see step 12). 



12 Distribute draft report for comment by paVties with a 
legitimate interest 



The draft should then be viewed by all those in any way 
affected by the review, and their comments invited, in writing, to 
react) the review team by a specified date. This step is not only a 
matter of principle, in line with the undertaking that there will be 
maximum consultation at all stages of the reviewj it is also a 
safeguard for the reviev/ team that they have not missed or 
misinterpreted something of significance, since any sins of 
commissmn or omission will certainty be drawn to their attention. 
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13 , Produce and distribute final report 



The review team will meet to consider reactions to its 
draft report and to decide on any changes which should be made. It 
will than issue its final report. Normally a review report has been 
agreed to by , all members of the review team. In exceptional 
circumstances, .dne or more of its members might produce a 
dissenting postscript or minority report, disagreeing with one or 
more points in the review findings or recommendations. Such 
dissension i,s however likely to damage the credibility and limit the 
effectiveness of the report. 



14 Initiate action on recommendations 



Normxally, Step 13 is the last step for which the revjew 
team has responsibility. The consequences of the report are, of 
course, crucial; in fact, in terms of getting 'value for money' from 
the review, they are a major justification for the whole exercise; 
but the steps required after the issuing of the report are usually 
taken by somebody else. These statements must be qualified by 
-^normally' and 'usually* because their truth depends upon the 
institution's authority structure as it relates to def)artmental 
reviews (see steps 1 and 2 above). Reviews are, of course, no 
substitute for the normal decision-making procedures of academic 
institutions, and can indeed feed into them. However, in certain 
circumstances, the review team may have some kind of monitoring 
responsibility or be the recipient of reports concerning 
implementation of its findings (see below). 

Action .on the review team's recommendations may be 
initiated in a number of different ways. The report may go to the 
chief executive and/or to the academic board and a request to the 
department to implement some or all of the recommendations may 
follow. It may go to chief executive and/or Sjpademic board simply 
for noting, with onus for initiating action remaining with the head 
of department or perhaps passing to the relevant dean or to one or 
other of the institution's standing committees. Either it is a 
matter of institution policy that review reports are acted upon, or 
someone must decide, for each review, that some or all of the 
recommendations are to be accepted and put into effect. 



Establish procedures for monitoring progress in 
implementation 



Action which has been initiated can peter out. A review 
team^s findings can get lost amid the welter of activity in a busy 
department unless there is some mechanism for ensuring the 
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continuation oYHactlon towards implementation. Progress may be 
reported to any one of a variety of individuals or committeeSy but 
it is vital that reports btb^ made aqd to a specified recipient. 
Whatever procedures for monitoring progress towards 
implementation are established, they should include target dates 
for the end of the whole exercise, i.e. dates when all 
recommendations which are to be implemented should have been 
put into effect. 
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IV CREATING AND ^ MAINTAINIMG AN APPROPRIATE 
CLIMATE FOR A REVIEW 



Much of the advice contained In this booklet, on the 
planning of a review, and on the steps which may be taken in 
carrying it out, has tHe important objective of creating and 
maintaining an appropriate climate; which, of course, means a 
favourable climate. 

This can be a formidable, task which all those involved in 
sponsoring, planning and conducting reviews should approach with 
continuing concern and forethought. Positive, constructive 
^ purposes have been' emphasised in 'this booklet, but it would be 
naive to suppose that protestations of such purposes would by 
themselves provide adequate' reassurance. A statement like *I want 
only to help you improve your nerformance* can be interpreted as 
•There is something wrong ^ ""h your performance. It needs 
Improving.' It may be seen as assuming, or (even- worse!) as about 
to make public, some deficiency in a department's performance. 

Statements about constructive purposes need to be 
translated into positive action or at least into firm commitments 
to positive action. Negative commitments - what the review panel 
will not do - are equally important: for example, that there are no 
hidden agendas, that significant information about the review 
procedures is not being withheld, that those who have given 
evidence privately will not be identified, that the review panel will 
not ignore comments on their draft report. 

Key features in creating and maintaining a favourable 
climate include the following; making purposes explicit; choosing a 
review team which has credibility (for example, not only with 
appropriate external members, but with members and/or a 
secretary thoroughly conversant with the politics and 
administrative practices of the institution); keeping all interested 
parties informed, with widespread- and genuine consultation; 
ensuring confidentiality wherever appropriate; providing 
opportunity for departmental comment before the report is 
finalised; and creating and fulfilling expectations that the report 
will be implemented. Unless department staff believe that 
something is likely to happen as a result of the review, the climate 
for the review will be poisoned by cynicism, which will adversely 
affect future reviews in that institution. On the other hand, the 
implementation of any one review may assist the climate of any 
future review if major emphasis is placed on the strengths of the 
department. It io important to take constructive action about 
weaknesses, but no less Important to build on strengths, and, in 
particular, to praise those whose performance is sound and who, 
because they are typically taken for granted, get little or no 
recognition of their substantial merits. In summary, the link 
between reviews and development, change and improvement should 
be recognised and continually stressed. 
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The acceptability of reviews is obviously enhanced if they 
are a normal feature of life in an institution; if, for example, it is 
institutional policy that all departments and sections arc reviewed 
at regular intervals. There is then no stigma attached to being 
reviewed) nobody need fedl discriminated against, everyone is 
being treated alil<e. 
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V IMPLICATIONS FOR INSTITUTIONAL POLICY 



In Ihis final section, key points for Institutions to consider 
in creating and maintaining an appropriate climate for reviews are 
listed. Since they also serve as a reminder of important issues 
referred to in Sections U and III, they may be read as a set of 
guidelines for institutional policy concerning departmental and' 
school reviews. 



Reviews should form part of the normal practices of 
educational Institutions, helping departments to chart their 
way forward and demonstrating the active self-scrutiny 
which the community expects. 



In establishing a system of reviews, an institution needs to 
provide leadership by 

I. 'selling' the concept of regular reviews to staff as a 
contribution to academic health of the institution 
and as an aid to rational planning - they are, after 
all, normal In many large Institutions; 

ii. educating staff as to the objectives, conduct and 
possible outcomes of such reviews; 

iii. providing the means by which any deficiencies 
identified in a review can be remedied. 



The institution needs to protect the rights of individual 
staff members by means of principles governing access to * 
information and issues of confidentiality such as the 
following: 

I. the purposes of the review, the areas to be 
examined and the terms of reference should be 
public, and staff of the department should be given 
an opportunity to comment on them before they are 
finalised; 

ii. all staff of the department should hav^ access to 
the review panel; 

ill. ell staff of the department should be provided with 
a list of the documentation considered by the panel; 

iv. the authorship of Information given to a review 
panel by individuals should be confidential to the 
panel; 
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Vf panel discussions should be confidential^to members 
of the panel; 

vi. although It is impossible to review a department's, 
performance without in .some sense reviewing the 
performance of the individuals who 'staff that 
department) no * evaluative comments concerning' 
individuals should appear in the review report; 

vii. members of the department should have an 
opportunity to comment on the report and its 
recommendations while it is still in draft form; 

vill. unless there are special and compelling reaspns 
against it, the report should be made available to all 
those with a legitimate interest. ; 



Revtevrpanets^shouttf 

i. be competent and impartial and be seen as such by 
the department under review; ' 

ii. contain at least one recognised expert in the 
relevant discipline from outside the institution^ and 
where appropriate a member of the relevant 
profession; 

iiU have ^ome continuity of membership, if possible) 
when several different departments are being 
reviewed; for ' example^ the same eenior 
administrative officer to service the panels and/or 
the same chairperson. * 



The institution' needs to establish guidelines for the 
conduct, reporting and implementation of reviews that will 
emphasise their constructive nature. These will be the 
principles on which review panels wii^' operatet in addition 
to those already set down in 3» above. They should include: 

i. the review panel must obtain relevant data to 
inform their judgment rather than making *data- 
free' recommendations; in particulari treating with 
extreme caution comments and opinions from 
anyone of whatever seniority, unless and until they 
can be supported by evidence; 
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the panel should take professional advice (if 

necessary) to ensure that the procedures are 

, appropriate to the time available and to the 
particular circumstances of the review; 

it should also consult with staff of the department 
being reviewed concerning procedures; 

iv. it sfiould be generous in the extent of notice given 
to the department to gather and provide the data it 
^ needs, and this may Include scope for a department 

' to conduct its own prior review, if that is the 
institution's policy; 

\ It should, however, complete the total review 
process. in the shortest time practicable so as not to 
lose momentum and credibility* 



The review panel should produce a report which 

I. ' contains cJearly stated recommendatior\s which can 
be implemented; 

ii« ciiscloses any limitations of the data on whifch 
recommendations are based, "and any^nmltations of 
the review itself; 

,iil. contains reasons for all comments and 
recommendations (including relevant data where 
available), so that the basis on which they are made 
is cle^^C^ 



The Institution and/or Its review panel can e^cpedite the 
implementation of recommendations by 

i* suggesting the means by which changes may be 
brought about; 

il. giving specific individuals responsibilities for 
' « changes; 

iil. defining an implementation period (say two years) 
with a follow-up at the end of the period and a 
report to an appropriate body. 
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